Arrangements detecting reset pci express bus in pci express path, and disabling use of pci express device

ABSTRACT

A root port connection functioning as a PCI express bridge, and having a PCI express path constituting a PCI express tree having a PCI express device or switch; when detecting a failure on a PCI express path, a PCI express device or switch transmits a failure signal; the root port transmits an SMI responsive to the failure signal; the CPU executes the BIOS responsive to the SMI; the BIOS collects a log of the PCI express path where failure is detected, analyzes the collected log to judge failure type, and upon a fatal failure on the PCI express path, resets the PCI express tree downstream of the root port that received the failure signal, and upon a non-fatal failure on the PCI express path, resets the PCI express device in which the failure occurred; and the CPU closes the reset PCI express device by executing the device driver.

CROSS REFERENCE TO RELATED APPLICATION

This is a continuation of U.S. application Ser. No. 12/685,760, filedJan. 12, 2010. This application relates to and claims priority fromJapanese Patent Application No. 2009-076274, filed on Mar. 26, 2009. Theentirety of the contents and subject matter of all of the above isincorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to electronic computers, and moreparticularly to computers which detect and handle a failure that mayoccur on a PCI express path.

2. Description of the Related Art

A typical computer system having a PCI express path includes a root portthat functions as a PCI express bridge (hereinafter, PCI Express isabbreviated to “PCIe”). The root port is connected to a CPU through aprimary bus, and is connected to a PCI express path through a secondarybus. The PCIe path forms a PCIe tree that includes a PCIe switch and aPCIe device which each are connected to the corresponding root port.

A failure which has occurred in the PCIe device is notified to the rootport through downstream and upstream ports of the PCIe switch. The rootport notifies the CPU of this failure by interrupting the CPU throughthe primary bus. A failure on another PCIe path including a PCIe switchis notified to the root port through a PCIe switch in which the failurehas been detected, or through a higher level PCIe switch connected tothe PCIe switch.

Incidentally, such techniques as described above are disclosed byJP-A-2004-348335 and JP-A-2005-196351 for example.

SUMMARY OF THE INVENTION

Heretofore, when a failure occurs on a PCIe path as described above anda CPU is then subject to interruption based on notification created inresponse to such a failure, the CPU has no choice but to reset a systemso as to reboot the operating system. In particular, a computer systemincludes a PCIe switch, and a plurality of computers, each of which isconnected to each of upstream ports of the PCIe switch. Such a computersystem has a problem that if a PCIe device shared by the plurality ofcomputers goes down, then the whole computer system will go down. Thismeans that the risk of system-down increases with the increase in thenumber of PCIe devices.

An object of the present invention is to disable only the use of thePCIe path on which a failure has occurred to thereby avoid resetting asystem.

According to one aspect of the present invention, there is provided acomputer comprising: a root port for detecting a failure on a PCIe path,and then for issuing a SMI (System Maintenance Interrupt) to a CPU; andthe CPU for, on the receipt of the SMI, executing BIOS to issue, throughthe root port, a PCIe reset to the PCIe path on which the failure hasoccurred.

According to the present invention, because only the use of the PCIepath on which a failure has occurred is disabled, system reset can beavoided.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a configuration of a computer systemaccording to this embodiment;

FIG. 2 is a diagram illustrating operation steps of individualmechanisms to be taken when a fatal failure is detected on a PCIe path;

FIG. 3 is a diagram illustrating operation steps of individualmechanisms to be taken when a non-fatal failure is detected on the PCIepath; and

FIG. 4 is a flowchart illustrating processing steps of a SMI handler ofBIOS.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiments of the present invention will be described below withreference to the accompanying drawings.

FIG. 1 is a diagram illustrating a configuration of a computer systemaccording to this embodiment. The computer system includes at least oneblade 1, multi-root IO virtualization PCIe switches (MR-IOV PCIe SW) 2,and PCIe devices 3. One blade 1 corresponds to one computer.

The blade 1 includes CPUs 11-1, 11-2, memories 12-1, 12-2, an IO hub 13,a Southbridge 15, a non-volatile memory 16, and a monitoring unit 18.

The IO hub 13 is connected to the CPUs 11-1, 11-2. The IO hub 13 is alsoconnected to the MR-IOV PCIe SW 2 or the PCIe device 3 through RPs (RootPort) 14 that each function as a PCIe bridge. The RPs 14 are configuredto be connected to the CPU 11 through a primary bus. In addition, alower route for the RPs 14 is associated with a PCIe path connectedthrough a secondary bus. The Southbridge 15 is connected to the IO hub13 through a 0-th RP 14 of the IO hub 13 and a DMI (Direct MediaInterface). The non-volatile memory 16 is connected to the Southbridge15. The non-volatile memory 16 stores a BIOS (Basic Input Output System)17. The monitoring unit 18 is connected to the IO hub 13 and monitorspossible failures in the CPUs 11-1, 11-2 and an I/O.

An operating system (OS) is loaded into the memories 12-1, 12-2, and isthen executed by the CPUs 11-1, 11-2. The BIOS 17 is copied to thememory 12-1, and is then executed by the CPUs 11-1, 11-2.

The MR-IOV PCIe SW 2 is connected to the specific RP 14 in the blade 1through an upstream port of the MR-IOV PCIe SW 2. The MR-IOV PCIe SW 2is also connected to another MR-IOV PCIe SW 2 or another PCIe device 3through a downstream port thereof. Blades 1 different from each othercan be connected to the upstream ports provided for each of the MR-IOVPCIe SWs 2.

FIG. 2 is a diagram illustrating operation steps of individualmechanisms to be taken when the MR-IOV PCIe SW 2 detects a fatal failureon the PCIe path. If the MR-IOV PCIe SW 2 detects a failure, then the RP14 which is connected to the MR-IOV PCIe SW 2 is notified of thefailure. The RP 14 transmits a signal indicating the fatal failure toboth the Southbridge 15 and the monitoring unit 18 through an ERR_N(2)pin. On the receipt of this signal, the monitoring unit 18 starts atimer. When the Southbridge 15 transmits a signal to prompt a SMI(System Maintenance Interrupt) to the RP 14 through a SMI# pin, the RP14 uses the SMI to notify the CPU 11 of the failure that has occurred onthe PCIe path. When the CPU 11 receives this interrupt, the control ispassed to the BIOS in the memory 12. The BIOS obtains a log of, forexample, a computing element included in the CPU 11, and also obtains alog of elements outside the CPU 11. The BIOS then analyzes the obtainedlogs to categorize failures into groups, before storing the logs in alog recording area. The BIOS stores the logs in the non-volatile memory16, or transmits them to the monitoring unit 18. In the case of afailure on the PCIe path, the BIOS transmits a PCIe reset signal throughthe RP 14 which is connected to the MR-IOV PCIe SW 2 that has detectedthe failure. Thus, a PCIe tree whose level is lower than the RP 14 thathas detected the failure is reset. Next, the BIOS transmits a timer stopsignal to the monitoring unit 18.

After the PCIe tree is reset, an OS device driver operates as follows.When a DMA termination interrupt or a DMA timeout is detected or when anIO access to the PCIe device 3 is made at the time of starting the IOaccess, the access to the PCIe device 3 which has been reset provides aMaster Abort response. Based on such a response, the OS device driverjudges that the PCIe device 3 under control cannot be used, andaccordingly, the device driver disables the use of the PCIe device 3. Ifthe system has a redundant configuration to cope with a failed device,the system can continue the operation thereof.

FIG. 3 is a diagram illustrating operation steps of individualmechanisms to be taken when the MR-IOV PCIe SW 2 or the PCIe device 3detects a non-fatal, non-recoverable failure on the PCIe path. If theMR-IOV PCIe SW 2 is involved in a failure, points of difference betweenthe operation steps of a non-fatal, non-recoverable failure and those ofa fatal failure are follows: the MR-IOV PCIe SW 2 notifies the RP 14 ofa failure through an ERR NONFATAL pin, and the RP 14 transmits a signalindicating the non-fatal, non-recoverable failure to both theSouthbridge 15 and the monitoring unit 18 through an ERR_N(1) pin. Inaddition, the BIOS transmits a secondary bus reset signal to a faileddevice through the MR-IOV PCIe SW 2 that has detected the failure. TheBIOS transmits a PCIe reset signal directly to the failed PCIe device 3in such a configuration that the PCIe device 3 is directly connected tothe RP 14 of the IO hub 13. The device driver detects the failed device,and disables the use of the failed device, in the same manner as theoperation steps taken when the fatal failure occurs.

FIG. 4 is a flowchart illustrating processing steps of a SMI handler ofthe BIOS. The SMI handler of the BIOS is started up in response to aSMI. Next, the SMI handler judges whether or not a failure has occurred(step 51). The BIOS reads a failure report register, which is includedin the CPU or the IO hub, to judge whether or not a PCIe path has failed(step 52). If it is judged that the PCIe path has failed, the BIOSobtains a general log for the failed PCIe path (step 53). If all the RPs14 in the IO hub 13 have not been inspected (step 54, NO), the BIOSjudges whether or not the remaining RP 14 is associated with a failurethat has been detected in the RP under inspection (step 55). If theremaining RP 14 is not judged to be associated with a failure that hasbeen detected in the RP under inspection, the process executed by theBIOS proceeds to a step 63. If the remaining RP 14 is judged to beassociated with a failure that has been detected in the RP underinspection, then the BIOS judges whether or not a DMI system has failed(step 56). If it is judged that the DMI system has failed, the processexecuted by the BIOS proceeds to a reboot step (step 57).

If it is not judged that the DMI system has failed, then the BIOS judgeswhether or not the interrupt has been executed based on a fatal failure(in other words, the BIOS judges whether or not this failure is a fatalfailure) (step 58). If it is judged that the interrupt has been executedbased on a fatal failure, the BIOS obtains a SEL (System Event Log) anda detailed log (step 59), and then issues a reset signal to a PCIe treewhose level is lower than the RP 14 in question (step 60). After that,the process proceeds to a step 63. If it is not judged that theinterrupt has been executed based on a fatal failure, the BIOS obtainsthe SEL and the detailed log (step 61), and then issues a reset signalto the failed device (step 62). After the processing of the step 55, 60or 62 ends, the BIOS increments a RP number by one (step 63), before theprocess returns to the step 54. If the inspection of all of the RPs 14of the IO hub 13 has been finished (step 54, YES), the BIOS ends theprocessing of the SMI handler.

The operation steps and processing steps described above are appliedalso to a computer system including a plurality of blades 1 in a similarmanner. In such a computer system, the RPs 14 of all the blades 1 thatare connected to the PCIe path are notified of a failure occurring onthe PCIe path. Each of the RPs 14 issues a SMI to the CPU 11 that isconnected through a primary bus. Each of the CPUs 11 receives this SMI,and executes the BIOS according to the processing steps so as to resetthe failed PCIe path. Each of the CPUs 11 executes a device driver inthe memory 12 so as to disable the use of the PCIe device in the failedPCIe path. Therefore, although the failed PCIe path, which is shared bythe plurality of blades 1, is disabled, the other PCIe paths can operatecontinuously.

The present invention is a method including the steps of: calling a BIOSin response to a SMI; allowing the BIOS to detect a failure on a PCIepath; resetting a PCIe tree or a PCIe device which has been detected;and allowing a device driver to indirectly detect a failed device and todisable the use of the failed device. In contrast, there is known amethod in which a MSI (Message Signal Interrupt) is applied to call adevice driver, and the device driver directly detects a failure on aPCIe path and recovers the failure. However, the SMI is advantageous inthat the SMI has a higher priority of interrupt acceptance than the MSI.Moreover, because the MSI is a memory write message, the MSI is appliedonly to a specific CPU 11, and the device driver disadvantageouslyexecutes processing more slowly than the BIOS. The method according tothe present invention has advantages that, in comparison with the methodin which a device driver detects a failure on a PCIe path, it ispossible to achieve a desired object without modifying OS and the devicedriver, and that the speed of failure detection is higher.

1. A method of handling a failure in a computer, the computercomprising: a memory that stores a BIOS and a device driver forcontrolling a PCI express device; a CPU that executes the BIOS and thedevice driver; and a root port connected to the CPU through a primarybus and to a PCI express path through a secondary bus, the root portfunctioning as a PCI express bridge; wherein: the PCI express pathconstitutes a PCI express tree having at least any one of the PCIexpress device and a PCI express switch; when detecting a failure on thePCI express path, the PCI express device or the PCI express switchtransmits a signal indicating the failure to the root port; the rootport transmits an SMI (System Maintenance Interrupt) to the CPU uponreceipt of the signal indicating the failure; the CPU executes the BIOSthat is started in response to receipt of the SMI; the BIOS collects alog of the PCI express path on which the failure has been detected, andanalyzes the collected log to judge the type of the failure, and whenthe failure is judged to be a fatal failure on the PCI express path bythe judgment, resets the PCI express tree downstream of the root portthat received the signal indicating the failure, and when the failure isjudged to be a non-fatal failure on the PCI express path by thejudgment, resets the PCI express device in which the failure occurred;and the CPU closes the reset PCI express device by executing the devicedriver.
 2. The method of handling a failure in a computer according toclaim 1, wherein: the memory storing the BIOS is connected to the rootport through a first interface; and when the failure is judged to be afailure on the first interface by the judgment, the BIOS reboots anoperating system which is executed by the CPU.
 3. The method of handlinga failure in a computer according to claim 2, wherein: the BIOS reads atleast any one of failure report registers in the CPU and in an IO hubhaving the root port and judges whether or not the failure is a failureon the PCI express path; and when the failure is judged to be a failureon the PCI express path, the BIOS collects the log of the PCI expresspath on which the failure has been detected.
 4. The method of handling afailure in a computer according to claim 3, wherein the BIOS stores thecollected log in a monitoring unit connected to the IO hub or in thememory.
 5. A computer comprising: a memory that stores a BIOS and adevice driver for controlling a PCI express device; a CPU that executesthe BIOS and the device driver; and a root port connected to the CPUthrough a primary bus and to a PCI express path through a secondary bus,the root port functioning as a PCI express bridge; wherein: the PCIexpress path constitutes a PCI express tree having at least any one ofthe PCI express device and a PCI express switch; when detecting afailure on the PCI express path, the PCI express device or the PCIexpress switch transmits a signal indicating the failure to the rootport; the root port transmits an SMI (System Maintenance Interrupt) tothe CPU upon receipt of the signal indicating the failure; the CPUexecutes the BIOS that is started in response to receipt of the SMI; theBIOS collects a log of the PCI express path on which the failure hasbeen detected, and analyzes the collected log to judge the type of thefailure, and when the failure is judged to be a fatal failure on the PCIexpress path by the judgment, resets the PCI express tree downstream ofthe root port that received the signal indicating the failure, and whenthe failure is judged to be a non-fatal failure on the PCI express pathby the judgment, resets the PCI express device in which the failureoccurred; and the CPU closes the reset PCI express device by executingthe device driver.
 6. The computer according to claim 5, wherein: thememory storing the BIOS is connected to the root port through a firstinterface; and when the failure is judged to be a failure on the firstinterface by the judgment, the BIOS reboots an operating system which isexecuted by the CPU.
 7. The computer according to claim 6, wherein: theBIOS reads at least any one of failure report registers in the CPU andin an IO hub having the root port, and judges whether or not the failureis a failure on the PCI express path; and when the failure is judged tobe a failure on the PCI express path, the BIOS collects the log of thePCI express path on which the failure has been detected.
 8. The computeraccording to claim 7, wherein the BIOS stores the collected log in amonitoring unit connected to the IO hub or in the memory.
 9. Thecomputer according to claim 8, wherein the first interface is a DirectMedia Interface.
 10. A computer system comprising: a plurality ofcomputers, each of the computers including a memory that stores a BIOSand a device driver for controlling a PCI express device; a CPU thatexecutes the BIOS and the device driver; and a root port connected tothe CPU through a primary bus and to a PCI express path through asecondary bus, the root port functioning as a PCI express bridge; andthe PCI express device and a PCI express switch that constitutes the PCIexpress path; wherein: when a failure on the PCI express path isdetected, the PCI express device or the PCI express switch transmits asignal indicating the failure to the root port; the root port, uponreceipt of the signal indicating the failure, transmits a signalindicating the failure to the root port of the computer connected to thePCI express path; the root port that received the signal indicating thefailure transmits an SMI (System Maintenance Interrupt) to the CPUconnected through the primary bus; the CPU executes the BIOS that isstarted in response to receipt of the SMI; the BIOS collects a log ofthe PCI express path on which the failure has been detected, andanalyzes the collected log to judge the type of the failure, and whenthe failure is judged to be a fatal failure on the PCI express path bythe judgment, resets the PCI express tree downstream of the root portthat received the signal indicating the failure, and when the failure isjudged to be a non-fatal failure on the PCI express path by thejudgment, resets the PCI express device in which the failure occurred;and the CPU closes the reset PCI express device by executing the devicedriver.
 11. The computer system according to claim 10, wherein: thememory storing the BIOS is connected to the root port through a firstinterface; and when the failure is judged to be a failure on the firstinterface by the judgment, the BIOS reboots an operating system which isexecuted by the CPU.
 12. The computer system according to claim 11,wherein: the BIOS reads at least any one of failure report registers inthe CPU and in an IO hub having the root port, and judges whether or notthe failure is a failure on the PCI express path; and when the failureis judged to be a failure on the PCI express path, the BIOS collects thelog of the PCI express path on which the failure has been detected. 13.The computer system according to claim 12, wherein the BIOS stores thecollected log in a monitoring unit connected to the IO hub or in thememory.
 14. The computer system according to claim 13, wherein the firstinterface is a Direct Media Interface.
 15. The computer system accordingto claim 14, wherein the computer is a server blade.